{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "85c30bdb",
   "metadata": {},
   "source": [
    "# Exploratory data analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "02d400cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from os.path import basename, exists\n",
    "\n",
    "\n",
    "def download(url):\n",
    "    filename = basename(url)\n",
    "    if not exists(filename):\n",
    "        from urllib.request import urlretrieve\n",
    "\n",
    "        local, _ = urlretrieve(url, filename)\n",
    "        print(\"Downloaded \" + local)\n",
    "\n",
    "\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/thinkstats2.py\")\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/thinkplot.py\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6ee58aea",
   "metadata": {},
   "outputs": [],
   "source": [
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/nsfg.py\")\n",
    "\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemPreg.dct\")\n",
    "download(\n",
    "    \"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemPreg.dat.gz\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ebc865a",
   "metadata": {},
   "source": [
    "The thesis of this book is that data combined with practical methods can\n",
    "answer questions and guide decisions under uncertainty.\n",
    "\n",
    "As an example, I present a case study motivated by a question I heard\n",
    "when my wife and I were expecting our first child: do first babies tend\n",
    "to arrive late?\n",
    "\n",
    "If you Google this question, you will find plenty of discussion. Some\n",
    "people claim it's true, others say it's a myth, and some people say it's\n",
    "the other way around: first babies come early.\n",
    "\n",
    "In many of these discussions, people provide data to support their\n",
    "claims. I found many examples like these:\n",
    "\n",
    "> \"My two friends that have given birth recently to their first babies,\n",
    "> BOTH went almost 2 weeks overdue before going into labour or being\n",
    "> induced.\"\n",
    ">\n",
    "> \"My first one came 2 weeks late and now I think the second one is\n",
    "> going to come out two weeks early!!\"\n",
    ">\n",
    "> \"I don't think that can be true because my sister was my mother's\n",
    "> first and she was early, as with many of my cousins.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69f30de7",
   "metadata": {},
   "source": [
    "Reports like these are called **anecdotal evidence** because they are\n",
    "based on data that is unpublished and usually personal. In casual\n",
    "conversation, there is nothing wrong with anecdotes, so I don't mean to\n",
    "pick on the people I quoted.\n",
    "\n",
    "But we might want evidence that is more persuasive and an answer that is\n",
    "more reliable. By those standards, anecdotal evidence usually fails,\n",
    "because:\n",
    "\n",
    "-   Small number of observations: If pregnancy length is longer for\n",
    "    first babies, the difference is probably small compared to natural\n",
    "    variation. In that case, we might have to compare a large number of\n",
    "    pregnancies to be sure that a difference exists.\n",
    "\n",
    "-   Selection bias: People who join a discussion of this question might\n",
    "    be interested because their first babies were late. In that case the\n",
    "    process of selecting data would bias the results.\n",
    "\n",
    "-   Confirmation bias: People who believe the claim might be more likely\n",
    "    to contribute examples that confirm it. People who doubt the claim\n",
    "    are more likely to cite counterexamples.\n",
    "\n",
    "-   Inaccuracy: Anecdotes are often personal stories, and often\n",
    "    misremembered, misrepresented, repeated inaccurately, etc.\n",
    "\n",
    "So how can we do better?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "136480d4",
   "metadata": {},
   "source": [
    "## A statistical approach\n",
    "\n",
    "To address the limitations of anecdotes, we will use the tools of\n",
    "statistics, which include:\n",
    "\n",
    "-   Data collection: We will use data from a large national survey that\n",
    "    was designed explicitly with the goal of generating statistically\n",
    "    valid inferences about the U.S. population.\n",
    "\n",
    "-   Descriptive statistics: We will generate statistics that summarize\n",
    "    the data concisely, and evaluate different ways to visualize data.\n",
    "\n",
    "-   Exploratory data analysis: We will look for patterns, differences,\n",
    "    and other features that address the questions we are interested in.\n",
    "    At the same time we will check for inconsistencies and identify\n",
    "    limitations.\n",
    "\n",
    "-   Estimation: We will use data from a sample to estimate\n",
    "    characteristics of the general population.\n",
    "\n",
    "-   Hypothesis testing: Where we see apparent effects, like a difference\n",
    "    between two groups, we will evaluate whether the effect might have\n",
    "    happened by chance.\n",
    "\n",
    "By performing these steps with care to avoid pitfalls, we can reach\n",
    "conclusions that are more justifiable and more likely to be correct."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41d329e0",
   "metadata": {},
   "source": [
    "## The National Survey of Family Growth\n",
    "\n",
    "Since 1973 the U.S. Centers for Disease Control and Prevention (CDC)\n",
    "have conducted the National Survey of Family Growth (NSFG), which is\n",
    "intended to gather \"information on family life, marriage and divorce,\n",
    "pregnancy, infertility, use of contraception, and men's and women's\n",
    "health. The survey results are used...to plan health services and health\n",
    "education programs, and to do statistical studies of families,\n",
    "fertility, and health.\" See <http://cdc.gov/nchs/nsfg.htm>.\n",
    "\n",
    "We will use data collected by this survey to investigate whether first\n",
    "babies tend to come late, and other questions. In order to use this data\n",
    "effectively, we have to understand the design of the study.\n",
    "\n",
    "The NSFG is a **cross-sectional** study, which means that it captures a\n",
    "snapshot of a group at a point in time. The most common alternative is a\n",
    "**longitudinal** study, which observes a group repeatedly over a period\n",
    "of time.\n",
    "\n",
    "The NSFG has been conducted seven times; each deployment is called a\n",
    "**cycle**. We will use data from Cycle 6, which was conducted from\n",
    "January 2002 to March 2003."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11a17e85",
   "metadata": {},
   "source": [
    "The goal of the survey is to draw conclusions about a **population**;\n",
    "the target population of the NSFG is people in the United States aged\n",
    "15-44. Ideally surveys would collect data from every member of the\n",
    "population, but that's seldom possible. Instead we collect data from a\n",
    "subset of the population called a **sample**. The people who participate\n",
    "in a survey are called **respondents**.\n",
    "\n",
    "In general, cross-sectional studies are meant to be **representative**,\n",
    "which means that every member of the target population has an equal\n",
    "chance of participating. That ideal is hard to achieve in practice, but\n",
    "people who conduct surveys come as close as they can.\n",
    "\n",
    "The NSFG is not representative; instead it is deliberately\n",
    "**oversampled**. The designers of the study recruited three\n",
    "groups---Hispanics, African-Americans and teenagers---at rates higher\n",
    "than their representation in the U.S. population, in order to make sure\n",
    "that the number of respondents in each of these groups is large enough\n",
    "to draw valid statistical inferences.\n",
    "\n",
    "Of course, the drawback of oversampling is that it is not as easy to\n",
    "draw conclusions about the general population based on statistics from\n",
    "the survey. We will come back to this point later."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5908f497",
   "metadata": {},
   "source": [
    "When working with this kind of data, it is important to be familiar with\n",
    "the **codebook**, which documents the design of the study, the survey\n",
    "questions, and the encoding of the responses. The codebook and user's\n",
    "guide for the NSFG data are available from\n",
    "<http://www.cdc.gov/nchs/nsfg/nsfg_cycle6.htm>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da1298b1",
   "metadata": {},
   "source": [
    "## Importing the data\n",
    "\n",
    "The code and data used in this book are available from\n",
    "<https://github.com/AllenDowney/ThinkStats2>. For information about\n",
    "downloading and working with this code, see\n",
    "Section [\\[code\\]](#code){reference-type=\"ref\" reference=\"code\"}.\n",
    "\n",
    "Once you download the code, you should have a file called\n",
    "`ThinkStats2/code/nsfg.py`. If you run it, it should read a data file,\n",
    "run some tests, and print a message like, \"All tests passed.\"\n",
    "\n",
    "Let's see what it does. Pregnancy data from Cycle 6 of the NSFG is in a\n",
    "file called `2002FemPreg.dat.gz`; it is a gzip-compressed data file in\n",
    "plain text (ASCII), with fixed width columns. Each line in the file is a\n",
    "**record** that contains data about one pregnancy.\n",
    "\n",
    "The format of the file is documented in `2002FemPreg.dct`, which is a\n",
    "Stata dictionary file. Stata is a statistical software system; a\n",
    "\"dictionary\" in this context is a list of variable names, types, and\n",
    "indices that identify where in each line to find each variable.\n",
    "\n",
    "For example, here are a few lines from `2002FemPreg.dct`:\n",
    "\n",
    "```\n",
    "infile dictionary {\n",
    "  _column(1)  str12  caseid    %12s  \"RESPONDENT ID NUMBER\"\n",
    "  _column(13) byte   pregordr   %2f  \"PREGNANCY ORDER (NUMBER)\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13928833",
   "metadata": {},
   "source": [
    "This dictionary describes two variables: `caseid` is a 12-character\n",
    "string that represents the respondent ID; `pregordr` is a one-byte\n",
    "integer that indicates which pregnancy this record describes for this\n",
    "respondent.\n",
    "\n",
    "The code you downloaded includes `thinkstats2.py`, which is a Python\n",
    "module that contains many classes and functions used in this book,\n",
    "including functions that read the Stata dictionary and the NSFG data\n",
    "file. Here's how they are used in `nsfg.py`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8ec05bc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from thinkstats2 import ReadStataDct\n",
    "\n",
    "def ReadFemPreg(dct_file='2002FemPreg.dct',\n",
    "                dat_file='2002FemPreg.dat.gz'):\n",
    "    dct = ReadStataDct(dct_file)\n",
    "    df = dct.ReadFixedWidth(dat_file, compression='gzip')\n",
    "    CleanFemPreg(df)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623703bf",
   "metadata": {},
   "source": [
    "`ReadStataDct` takes the name of the dictionary file and returns `dct`,\n",
    "a `FixedWidthVariables` object that contains the information from the\n",
    "dictionary file. `dct` provides `ReadFixedWidth`, which reads the data\n",
    "file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f164c4b",
   "metadata": {},
   "source": [
    "## DataFrames\n",
    "\n",
    "The result of `ReadFixedWidth` is a DataFrame, which is the fundamental\n",
    "data structure provided by pandas, which is a Python data and statistics\n",
    "package we'll use throughout this book. A DataFrame contains a row for\n",
    "each record, in this case one row per pregnancy, and a column for each\n",
    "variable.\n",
    "\n",
    "In addition to the data, a DataFrame also contains the variable names\n",
    "and their types, and it provides methods for accessing and modifying the\n",
    "data.\n",
    "\n",
    "If you print `preg` you get a truncated view of the rows and columns, and\n",
    "the shape of the DataFrame, which is 13593 rows/records and 244\n",
    "columns/variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "87e0bfbc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/downey/ThinkStats/soln/nsfg.py:68: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  df.birthwgt_lb.replace(na_vals, np.nan, inplace=True)\n",
      "/home/downey/ThinkStats/soln/nsfg.py:69: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  df.birthwgt_oz.replace(na_vals, np.nan, inplace=True)\n",
      "/home/downey/ThinkStats/soln/nsfg.py:70: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  df.hpagelb.replace(na_vals, np.nan, inplace=True)\n",
      "/home/downey/ThinkStats/soln/nsfg.py:72: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  df.babysex.replace([7, 9], np.nan, inplace=True)\n",
      "/home/downey/ThinkStats/soln/nsfg.py:73: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.\n",
      "The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.\n",
      "\n",
      "For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.\n",
      "\n",
      "\n",
      "  df.nbrnaliv.replace([9], np.nan, inplace=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>caseid</th>\n",
       "      <th>pregordr</th>\n",
       "      <th>howpreg_n</th>\n",
       "      <th>howpreg_p</th>\n",
       "      <th>moscurrp</th>\n",
       "      <th>nowprgdk</th>\n",
       "      <th>pregend1</th>\n",
       "      <th>pregend2</th>\n",
       "      <th>nbrnaliv</th>\n",
       "      <th>multbrth</th>\n",
       "      <th>...</th>\n",
       "      <th>laborfor_i</th>\n",
       "      <th>religion_i</th>\n",
       "      <th>metro_i</th>\n",
       "      <th>basewgt</th>\n",
       "      <th>adj_mod_basewgt</th>\n",
       "      <th>finalwgt</th>\n",
       "      <th>secu_p</th>\n",
       "      <th>sest</th>\n",
       "      <th>cmintvw</th>\n",
       "      <th>totalwgt_lb</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3410.389399</td>\n",
       "      <td>3869.349602</td>\n",
       "      <td>6448.271112</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.8125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3410.389399</td>\n",
       "      <td>3869.349602</td>\n",
       "      <td>6448.271112</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.8750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7226.301740</td>\n",
       "      <td>8567.549110</td>\n",
       "      <td>12999.542264</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.1250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7226.301740</td>\n",
       "      <td>8567.549110</td>\n",
       "      <td>12999.542264</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7226.301740</td>\n",
       "      <td>8567.549110</td>\n",
       "      <td>12999.542264</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.1875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13588</th>\n",
       "      <td>12571</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4670.540953</td>\n",
       "      <td>5795.692880</td>\n",
       "      <td>6269.200989</td>\n",
       "      <td>1</td>\n",
       "      <td>78</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.1875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13589</th>\n",
       "      <td>12571</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4670.540953</td>\n",
       "      <td>5795.692880</td>\n",
       "      <td>6269.200989</td>\n",
       "      <td>1</td>\n",
       "      <td>78</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13590</th>\n",
       "      <td>12571</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4670.540953</td>\n",
       "      <td>5795.692880</td>\n",
       "      <td>6269.200989</td>\n",
       "      <td>1</td>\n",
       "      <td>78</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13591</th>\n",
       "      <td>12571</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4670.540953</td>\n",
       "      <td>5795.692880</td>\n",
       "      <td>6269.200989</td>\n",
       "      <td>1</td>\n",
       "      <td>78</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13592</th>\n",
       "      <td>12571</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4670.540953</td>\n",
       "      <td>5795.692880</td>\n",
       "      <td>6269.200989</td>\n",
       "      <td>1</td>\n",
       "      <td>78</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.5000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13593 rows × 244 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       caseid  pregordr  howpreg_n  howpreg_p  moscurrp  nowprgdk  pregend1  \\\n",
       "0           1         1        NaN        NaN       NaN       NaN       6.0   \n",
       "1           1         2        NaN        NaN       NaN       NaN       6.0   \n",
       "2           2         1        NaN        NaN       NaN       NaN       5.0   \n",
       "3           2         2        NaN        NaN       NaN       NaN       6.0   \n",
       "4           2         3        NaN        NaN       NaN       NaN       6.0   \n",
       "...       ...       ...        ...        ...       ...       ...       ...   \n",
       "13588   12571         1        NaN        NaN       NaN       NaN       6.0   \n",
       "13589   12571         2        NaN        NaN       NaN       NaN       3.0   \n",
       "13590   12571         3        NaN        NaN       NaN       NaN       3.0   \n",
       "13591   12571         4        NaN        NaN       NaN       NaN       6.0   \n",
       "13592   12571         5        NaN        NaN       NaN       NaN       6.0   \n",
       "\n",
       "       pregend2  nbrnaliv  multbrth  ...  laborfor_i  religion_i  metro_i  \\\n",
       "0           NaN       1.0       NaN  ...           0           0        0   \n",
       "1           NaN       1.0       NaN  ...           0           0        0   \n",
       "2           NaN       3.0       5.0  ...           0           0        0   \n",
       "3           NaN       1.0       NaN  ...           0           0        0   \n",
       "4           NaN       1.0       NaN  ...           0           0        0   \n",
       "...         ...       ...       ...  ...         ...         ...      ...   \n",
       "13588       NaN       1.0       NaN  ...           0           0        0   \n",
       "13589       NaN       NaN       NaN  ...           0           0        0   \n",
       "13590       NaN       NaN       NaN  ...           0           0        0   \n",
       "13591       NaN       1.0       NaN  ...           0           0        0   \n",
       "13592       NaN       1.0       NaN  ...           0           0        0   \n",
       "\n",
       "           basewgt  adj_mod_basewgt      finalwgt  secu_p  sest  cmintvw  \\\n",
       "0      3410.389399      3869.349602   6448.271112       2     9      NaN   \n",
       "1      3410.389399      3869.349602   6448.271112       2     9      NaN   \n",
       "2      7226.301740      8567.549110  12999.542264       2    12      NaN   \n",
       "3      7226.301740      8567.549110  12999.542264       2    12      NaN   \n",
       "4      7226.301740      8567.549110  12999.542264       2    12      NaN   \n",
       "...            ...              ...           ...     ...   ...      ...   \n",
       "13588  4670.540953      5795.692880   6269.200989       1    78      NaN   \n",
       "13589  4670.540953      5795.692880   6269.200989       1    78      NaN   \n",
       "13590  4670.540953      5795.692880   6269.200989       1    78      NaN   \n",
       "13591  4670.540953      5795.692880   6269.200989       1    78      NaN   \n",
       "13592  4670.540953      5795.692880   6269.200989       1    78      NaN   \n",
       "\n",
       "       totalwgt_lb  \n",
       "0           8.8125  \n",
       "1           7.8750  \n",
       "2           9.1250  \n",
       "3           7.0000  \n",
       "4           6.1875  \n",
       "...            ...  \n",
       "13588       6.1875  \n",
       "13589          NaN  \n",
       "13590          NaN  \n",
       "13591       7.5000  \n",
       "13592       7.5000  \n",
       "\n",
       "[13593 rows x 244 columns]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nsfg\n",
    "preg = nsfg.ReadFemPreg()\n",
    "preg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91e89750",
   "metadata": {},
   "source": [
    "The DataFrame is too big to display, so the output is truncated. The\n",
    "last line reports the number of rows and columns.\n",
    "\n",
    "The attribute `columns` returns a sequence of column names as Unicode\n",
    "strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b3e50782",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['caseid', 'pregordr', 'howpreg_n', 'howpreg_p', 'moscurrp', 'nowprgdk',\n",
       "       'pregend1', 'pregend2', 'nbrnaliv', 'multbrth',\n",
       "       ...\n",
       "       'laborfor_i', 'religion_i', 'metro_i', 'basewgt', 'adj_mod_basewgt',\n",
       "       'finalwgt', 'secu_p', 'sest', 'cmintvw', 'totalwgt_lb'],\n",
       "      dtype='object', length=244)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preg.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a396756",
   "metadata": {},
   "source": [
    "The result is an Index, which is another pandas data structure. We'll\n",
    "learn more about Index later, but for now we'll treat it like a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3ca1b9a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pregordr'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preg.columns[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e82ef6e6",
   "metadata": {},
   "source": [
    "To access a column from a DataFrame, you can use the column name as a\n",
    "key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e7dbfce1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pregordr = preg['pregordr']\n",
    "type(pregordr)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aad9af2d",
   "metadata": {},
   "source": [
    "The result is a Series, yet another pandas data structure. A Series is\n",
    "like a Python list with some additional features. When you print a\n",
    "Series, you get the indices and the corresponding values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "189b4869",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "1        2\n",
       "2        1\n",
       "3        2\n",
       "4        3\n",
       "        ..\n",
       "13588    1\n",
       "13589    2\n",
       "13590    3\n",
       "13591    4\n",
       "13592    5\n",
       "Name: pregordr, Length: 13593, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pregordr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1862cdfe",
   "metadata": {},
   "source": [
    "In this example the indices are integers from 0 to 13592, but in general\n",
    "they can be any sortable type. The elements are also integers, but they\n",
    "can be any type.\n",
    "\n",
    "The last line includes the variable name, Series length, and data type;\n",
    "`int64` is one of the types provided by NumPy. If you run this example\n",
    "on a 32-bit machine you might see `int32`.\n",
    "\n",
    "You can access the elements of a Series using integer indices and\n",
    "slices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b2b489c3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pregordr[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "480195c8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    1\n",
       "3    2\n",
       "4    3\n",
       "Name: pregordr, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pregordr[2:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60f4cf6e",
   "metadata": {},
   "source": [
    "The result of the index operator is an `int64`; the result of the slice\n",
    "is another Series.\n",
    "\n",
    "You can also access the columns of a DataFrame using dot notation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "30be7d51",
   "metadata": {},
   "outputs": [],
   "source": [
    "pregordr = preg.pregordr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004fc389",
   "metadata": {},
   "source": [
    "This notation only works if the column name is a valid Python\n",
    "identifier, so it has to begin with a letter, can't contain spaces, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01847dd9",
   "metadata": {},
   "source": [
    "## Variables\n",
    "\n",
    "We have already seen two variables in the NSFG dataset, `caseid` and\n",
    "`pregordr`, and we have seen that there are 244 variables in total. For\n",
    "the explorations in this book, I use the following variables:\n",
    "\n",
    "-   `caseid` is the integer ID of the respondent.\n",
    "\n",
    "-   `prglngth` is the integer duration of the pregnancy in weeks.\n",
    "\n",
    "-   `outcome` is an integer code for the outcome of the pregnancy. The\n",
    "    code 1 indicates a live birth.\n",
    "\n",
    "-   `pregordr` is a pregnancy serial number; for example, the code for a\n",
    "    respondent's first pregnancy is 1, for the second pregnancy is 2,\n",
    "    and so on.\n",
    "\n",
    "-   `birthord` is a serial number for live births; the code for a\n",
    "    respondent's first child is 1, and so on. For outcomes other than\n",
    "    live birth, this field is blank.\n",
    "\n",
    "-   `birthwgt_lb` and `birthwgt_oz` contain the pounds and ounces parts\n",
    "    of the birth weight of the baby.\n",
    "\n",
    "-   `agepreg` is the mother's age at the end of the pregnancy.\n",
    "\n",
    "-   `finalwgt` is the statistical weight associated with the respondent.\n",
    "    It is a floating-point value that indicates the number of people in\n",
    "    the U.S. population this respondent represents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37c86aa9",
   "metadata": {},
   "source": [
    "If you read the codebook carefully, you will see that many of the\n",
    "variables are **recodes**, which means that they are not part of the\n",
    "**raw data** collected by the survey; they are calculated using the raw\n",
    "data.\n",
    "\n",
    "For example, `prglngth` for live births is equal to the raw variable\n",
    "`wksgest` (weeks of gestation) if it is available; otherwise it is\n",
    "estimated using `mosgest * 4.33` (months of gestation times the average\n",
    "number of weeks in a month).\n",
    "\n",
    "Recodes are often based on logic that checks the consistency and\n",
    "accuracy of the data. In general it is a good idea to use recodes when\n",
    "they are available, unless there is a compelling reason to process the\n",
    "raw data yourself."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdf027c2",
   "metadata": {},
   "source": [
    "## Transformation\n",
    "\n",
    "When you import data like this, you often have to check for errors, deal\n",
    "with special values, convert data into different formats, and perform\n",
    "calculations. These operations are called **data cleaning**.\n",
    "\n",
    "`nsfg.py` includes `CleanFemPreg`, a function that cleans the variables\n",
    "I am planning to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9f3c9f8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def CleanFemPreg(df):\n",
    "    df.agepreg /= 100.0\n",
    "\n",
    "    na_vals = [97, 98, 99]\n",
    "    df['birthwgt_lb'] = df.birthwgt_lb.replace(na_vals, np.nan)\n",
    "    df['birthwgt_oz'] = df.birthwgt_oz.replace(na_vals, np.nan)\n",
    "\n",
    "    df['totalwgt_lb'] = df.birthwgt_lb + df.birthwgt_oz / 16.0    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdeeec8f",
   "metadata": {},
   "source": [
    "`agepreg` contains the mother's age at the end of the pregnancy. In the\n",
    "data file, `agepreg` is encoded as an integer number of centiyears. So\n",
    "the first line divides each element of `agepreg` by 100, yielding a\n",
    "floating-point value in years.\n",
    "\n",
    "`birthwgt_lb` and `birthwgt_oz` contain the weight of the baby, in\n",
    "pounds and ounces, for pregnancies that end in live birth. In addition\n",
    "it uses several special codes:"
   ]
  },
  {
   "cell_type": "raw",
   "id": "a5ba7cc4",
   "metadata": {},
   "source": [
    "97      NOT ASCERTAINED\n",
    "98      REFUSED\n",
    "99      DON'T KNOW"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7fedf29",
   "metadata": {},
   "source": [
    "Special values encoded as numbers are *dangerous* because if they are\n",
    "not handled properly, they can generate bogus results, like a 99-pound\n",
    "baby. The `replace` method replaces these values with `np.nan`, a\n",
    "special floating-point value that represents \"not a number.\"\n",
    "\n",
    "As part of the IEEE floating-point standard, all mathematical operations\n",
    "return `nan` if either argument is `nan`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "017961a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.nan / 100.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3f5f97b",
   "metadata": {},
   "source": [
    "So computations with `nan` tend to do the right thing, and most pandas\n",
    "functions handle `nan` appropriately. But dealing with missing data will\n",
    "be a recurring issue.\n",
    "\n",
    "The last line of `CleanFemPreg` creates a new column `totalwgt_lb` that\n",
    "combines pounds and ounces into a single quantity, in pounds.\n",
    "\n",
    "One important note: when you add a new column to a DataFrame, you must\n",
    "use dictionary syntax, like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6376386f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# CORRECT\n",
    "preg['totalwgt_lb'] = preg.birthwgt_lb + preg.birthwgt_oz / 16.0 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f37b46a",
   "metadata": {},
   "source": [
    "Not dot notation, like this:"
   ]
  },
  {
   "cell_type": "raw",
   "id": "ee7b53a0",
   "metadata": {},
   "source": [
    "# WRONG!\n",
    "preg.totalwgt_lb = preg.birthwgt_lb + preg.birthwgt_oz / 16.0 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1466aba3",
   "metadata": {},
   "source": [
    "The version with dot notation adds an attribute to the DataFrame object,\n",
    "but that attribute is not treated as a new column."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca667a83",
   "metadata": {},
   "source": [
    "## Validation\n",
    "\n",
    "When data is exported from one software environment and imported into\n",
    "another, errors might be introduced. And when you are getting familiar\n",
    "with a new dataset, you might interpret data incorrectly or introduce\n",
    "other misunderstandings. If you take time to validate the data, you can\n",
    "save time later and avoid errors.\n",
    "\n",
    "One way to validate data is to compute basic statistics and compare them\n",
    "with published results. For example, the NSFG codebook includes tables\n",
    "that summarize each variable. Here is the table for `outcome`, which\n",
    "encodes the outcome of each pregnancy:"
   ]
  },
  {
   "cell_type": "raw",
   "id": "ea478208",
   "metadata": {},
   "source": [
    "value   label                  Total\n",
    "1       LIVE BIRTH              9148\n",
    "2       INDUCED ABORTION        1862\n",
    "3       STILLBIRTH               120\n",
    "4       MISCARRIAGE             1921\n",
    "5       ECTOPIC PREGNANCY        190\n",
    "6       CURRENT PREGNANCY        352"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb9d01a0",
   "metadata": {},
   "source": [
    "The Series class provides a method, `value_counts`, that counts the\n",
    "number of times each value appears. If we select the `outcome` Series\n",
    "from the DataFrame, we can use `value_counts` to compare with the\n",
    "published data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "61fc2a3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "outcome\n",
       "1    9148\n",
       "2    1862\n",
       "3     120\n",
       "4    1921\n",
       "5     190\n",
       "6     352\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preg.outcome.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d57f49ea",
   "metadata": {},
   "source": [
    "The result of `value_counts` is a Series; `sort_index()` sorts the\n",
    "Series by index, so the values appear in order.\n",
    "\n",
    "Comparing the results with the published table, it looks like the values\n",
    "in `outcome` are correct. Similarly, here is the published table for\n",
    "`birthwgt_lb`"
   ]
  },
  {
   "cell_type": "raw",
   "id": "a6771a9d",
   "metadata": {},
   "source": [
    "value   label                  Total\n",
    ".       INAPPLICABLE            4449\n",
    "0-5     UNDER 6 POUNDS          1125\n",
    "6       6 POUNDS                2223\n",
    "7       7 POUNDS                3049\n",
    "8       8 POUNDS                1889\n",
    "9-95    9 POUNDS OR MORE         799"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afccd71b",
   "metadata": {},
   "source": [
    "And here are the value counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c2cdb474",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "birthwgt_lb\n",
       "0.0        8\n",
       "1.0       40\n",
       "2.0       53\n",
       "3.0       98\n",
       "4.0      229\n",
       "5.0      697\n",
       "6.0     2223\n",
       "7.0     3049\n",
       "8.0     1889\n",
       "9.0      623\n",
       "10.0     132\n",
       "11.0      26\n",
       "12.0      10\n",
       "13.0       3\n",
       "14.0       3\n",
       "15.0       1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preg.birthwgt_lb.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca21655d",
   "metadata": {},
   "source": [
    "The counts for 6, 7, and 8 pounds check out, and if you add up the\n",
    "counts for 0-5 and 9-95, they check out, too. But if you look more\n",
    "closely, you will notice one value that has to be an error, a 51 pound\n",
    "baby!\n",
    "\n",
    "To deal with this error, I added a line to `CleanFemPreg`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bc5c2351",
   "metadata": {},
   "outputs": [],
   "source": [
    "preg.loc[preg.birthwgt_lb > 20, 'birthwgt_lb'] = np.nan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cab151e6",
   "metadata": {},
   "source": [
    "This statement replaces invalid values with `np.nan`. The attribute\n",
    "`loc` provides several ways to select rows and columns from a DataFrame.\n",
    "In this example, the first expression in brackets is the row indexer;\n",
    "the second expression selects the column.\n",
    "\n",
    "The expression `preg.birthwgt_lb > 20` yields a Series of type `bool`,\n",
    "where True indicates that the condition is true. When a boolean Series\n",
    "is used as an index, it selects only the elements that satisfy the\n",
    "condition."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4234e27e",
   "metadata": {},
   "source": [
    "## Interpretation\n",
    "\n",
    "To work with data effectively, you have to think on two levels at the\n",
    "same time: the level of statistics and the level of context.\n",
    "\n",
    "As an example, let's look at the sequence of outcomes for a few\n",
    "respondents. Because of the way the data files are organized, we have to\n",
    "do some processing to collect the pregnancy data for each respondent.\n",
    "Here's a function that does that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "aa1a58ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "def MakePregMap(df):\n",
    "    d = defaultdict(list)\n",
    "    for index, caseid in df.caseid.iteritems():\n",
    "        d[caseid].append(index)\n",
    "    return d"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c77c46f",
   "metadata": {},
   "source": [
    "`df` is the DataFrame with pregnancy data. The `iteritems` method\n",
    "enumerates the index (row number) and `caseid` for each pregnancy.\n",
    "\n",
    "`d` is a dictionary that maps from each case ID to a list of indices. If\n",
    "you are not familiar with `defaultdict`, it is in the Python\n",
    "`collections` module. Using `d`, we can look up a respondent and get the\n",
    "indices of that respondent's pregnancies.\n",
    "\n",
    "This example looks up one respondent and prints a list of outcomes for\n",
    "her pregnancies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e4d4907d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 4, 4, 4, 4, 4, 1])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "caseid = 10229\n",
    "preg_map = nsfg.MakePregMap(df)\n",
    "indices = preg_map[caseid]\n",
    "df.outcome[indices].values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c5ef31b",
   "metadata": {},
   "source": [
    "`indices` is the list of indices for pregnancies corresponding to\n",
    "respondent `10229`.\n",
    "\n",
    "Using this list as an index into `df.outcome` selects the indicated rows\n",
    "and yields a Series. Instead of printing the whole Series, I selected\n",
    "the `values` attribute, which is a NumPy array.\n",
    "\n",
    "The outcome code `1` indicates a live birth. Code `4` indicates a\n",
    "miscarriage; that is, a pregnancy that ended spontaneously, usually with\n",
    "no known medical cause.\n",
    "\n",
    "Statistically this respondent is not unusual. Miscarriages are common\n",
    "and there are other respondents who reported as many or more.\n",
    "\n",
    "But remembering the context, this data tells the story of a woman who\n",
    "was pregnant six times, each time ending in miscarriage. Her seventh and\n",
    "most recent pregnancy ended in a live birth. If we consider this data\n",
    "with empathy, it is natural to be moved by the story it tells.\n",
    "\n",
    "Each record in the NSFG dataset represents a person who provided honest\n",
    "answers to many personal and difficult questions. We can use this data\n",
    "to answer statistical questions about family life, reproduction, and\n",
    "health. At the same time, we have an obligation to consider the people\n",
    "represented by the data, and to afford them respect and gratitude."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b3f8c2a",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd7b7186",
   "metadata": {},
   "source": [
    "Select the `birthord` column, print the value counts, and compare to results published in the [codebook](https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NSFG/Cycle6Codebook-Pregnancy.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "56d89965",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "birthord\n",
       "1.0     4413\n",
       "2.0     2874\n",
       "3.0     1234\n",
       "4.0      421\n",
       "5.0      126\n",
       "6.0       50\n",
       "7.0       20\n",
       "8.0        7\n",
       "9.0        2\n",
       "10.0       1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "preg.birthord.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e99e51a6",
   "metadata": {},
   "source": [
    "We can also use `isnull` to count the number of nans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d5c78cde",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4445"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preg.birthord.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90fd57dd",
   "metadata": {},
   "source": [
    "Select the `prglngth` column, print the value counts, and compare to results published in the [codebook](https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NSFG/Cycle6Codebook-Pregnancy.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "31e19635",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "prglngth\n",
       "0       15\n",
       "1        9\n",
       "2       78\n",
       "3      151\n",
       "4      412\n",
       "5      181\n",
       "6      543\n",
       "7      175\n",
       "8      409\n",
       "9      594\n",
       "10     137\n",
       "11     202\n",
       "12     170\n",
       "13     446\n",
       "14      29\n",
       "15      39\n",
       "16      44\n",
       "17     253\n",
       "18      17\n",
       "19      34\n",
       "20      18\n",
       "21      37\n",
       "22     147\n",
       "23      12\n",
       "24      31\n",
       "25      15\n",
       "26     117\n",
       "27       8\n",
       "28      38\n",
       "29      23\n",
       "30     198\n",
       "31      29\n",
       "32     122\n",
       "33      50\n",
       "34      60\n",
       "35     357\n",
       "36     329\n",
       "37     457\n",
       "38     609\n",
       "39    4744\n",
       "40    1120\n",
       "41     591\n",
       "42     328\n",
       "43     148\n",
       "44      46\n",
       "45      10\n",
       "46       1\n",
       "47       1\n",
       "48       7\n",
       "50       2\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "preg.prglngth.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4deb5975",
   "metadata": {},
   "source": [
    "To compute the mean of a column, you can invoke the `mean` method on a Series.  For example, here is the mean birthweight in pounds:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e455ff63",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.265628457623368"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preg.totalwgt_lb.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a6aabaa",
   "metadata": {},
   "source": [
    "Create a new column named <tt>totalwgt_kg</tt> that contains birth weight in kilograms.  Compute its mean.  Remember that when you create a new column, you have to use dictionary syntax, not dot notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "009d0f95",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.302558389828803"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "preg['totalwgt_kg'] = preg.totalwgt_lb / 2.2\n",
    "preg.totalwgt_kg.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f290fa",
   "metadata": {},
   "source": [
    "`nsfg.py` also provides `ReadFemResp`, which reads the female respondents file and returns a `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "7a6efd3f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded 2002FemResp.dct\n",
      "Downloaded 2002FemResp.dat.gz\n"
     ]
    }
   ],
   "source": [
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemResp.dct\")\n",
    "download(\"https://github.com/AllenDowney/ThinkStats2/raw/master/code/2002FemResp.dat.gz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "1998580f",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = nsfg.ReadFemResp()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d02885c",
   "metadata": {},
   "source": [
    "`DataFrame` provides a method `head` that displays the first five rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "9c9929b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>caseid</th>\n",
       "      <th>rscrinf</th>\n",
       "      <th>rdormres</th>\n",
       "      <th>rostscrn</th>\n",
       "      <th>rscreenhisp</th>\n",
       "      <th>rscreenrace</th>\n",
       "      <th>age_a</th>\n",
       "      <th>age_r</th>\n",
       "      <th>cmbirth</th>\n",
       "      <th>agescrn</th>\n",
       "      <th>...</th>\n",
       "      <th>pubassis_i</th>\n",
       "      <th>basewgt</th>\n",
       "      <th>adj_mod_basewgt</th>\n",
       "      <th>finalwgt</th>\n",
       "      <th>secu_r</th>\n",
       "      <th>sest</th>\n",
       "      <th>cmintvw</th>\n",
       "      <th>cmlstyr</th>\n",
       "      <th>screentime</th>\n",
       "      <th>intvlngth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2298</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>902</td>\n",
       "      <td>27</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3247.916977</td>\n",
       "      <td>5123.759559</td>\n",
       "      <td>5556.717241</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>1234</td>\n",
       "      <td>1222</td>\n",
       "      <td>18:26:36</td>\n",
       "      <td>110.492667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5012</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>42</td>\n",
       "      <td>42</td>\n",
       "      <td>718</td>\n",
       "      <td>42</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>2335.279149</td>\n",
       "      <td>2846.799490</td>\n",
       "      <td>4744.191350</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>1233</td>\n",
       "      <td>1221</td>\n",
       "      <td>16:30:59</td>\n",
       "      <td>64.294000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11586</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>708</td>\n",
       "      <td>43</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>2335.279149</td>\n",
       "      <td>2846.799490</td>\n",
       "      <td>4744.191350</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>1234</td>\n",
       "      <td>1222</td>\n",
       "      <td>18:19:09</td>\n",
       "      <td>75.149167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6794</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>1042</td>\n",
       "      <td>15</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3783.152221</td>\n",
       "      <td>5071.464231</td>\n",
       "      <td>5923.977368</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>1234</td>\n",
       "      <td>1222</td>\n",
       "      <td>15:54:43</td>\n",
       "      <td>28.642833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>616</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>991</td>\n",
       "      <td>20</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>5341.329968</td>\n",
       "      <td>6437.335772</td>\n",
       "      <td>7229.128072</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>1233</td>\n",
       "      <td>1221</td>\n",
       "      <td>14:19:44</td>\n",
       "      <td>69.502667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 3087 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   caseid  rscrinf  rdormres  rostscrn  rscreenhisp  rscreenrace  age_a  \\\n",
       "0    2298        1         5         5            1          5.0     27   \n",
       "1    5012        1         5         1            5          5.0     42   \n",
       "2   11586        1         5         1            5          5.0     43   \n",
       "3    6794        5         5         4            1          5.0     15   \n",
       "4     616        1         5         4            1          5.0     20   \n",
       "\n",
       "   age_r  cmbirth  agescrn  ...  pubassis_i      basewgt  adj_mod_basewgt  \\\n",
       "0     27      902       27  ...           0  3247.916977      5123.759559   \n",
       "1     42      718       42  ...           0  2335.279149      2846.799490   \n",
       "2     43      708       43  ...           0  2335.279149      2846.799490   \n",
       "3     15     1042       15  ...           0  3783.152221      5071.464231   \n",
       "4     20      991       20  ...           0  5341.329968      6437.335772   \n",
       "\n",
       "      finalwgt  secu_r  sest  cmintvw  cmlstyr  screentime   intvlngth  \n",
       "0  5556.717241       2    18     1234     1222    18:26:36  110.492667  \n",
       "1  4744.191350       2    18     1233     1221    16:30:59   64.294000  \n",
       "2  4744.191350       2    18     1234     1222    18:19:09   75.149167  \n",
       "3  5923.977368       2    18     1234     1222    15:54:43   28.642833  \n",
       "4  7229.128072       2    18     1233     1221    14:19:44   69.502667  \n",
       "\n",
       "[5 rows x 3087 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3cf1ff0",
   "metadata": {},
   "source": [
    "Select the `age_r` column from `resp` and print the value counts.  How old are the youngest and oldest respondents?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "13b80162",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age_r\n",
       "15    217\n",
       "16    223\n",
       "17    234\n",
       "18    235\n",
       "19    241\n",
       "20    258\n",
       "21    267\n",
       "22    287\n",
       "23    282\n",
       "24    269\n",
       "25    267\n",
       "26    260\n",
       "27    255\n",
       "28    252\n",
       "29    262\n",
       "30    292\n",
       "31    278\n",
       "32    273\n",
       "33    257\n",
       "34    255\n",
       "35    262\n",
       "36    266\n",
       "37    271\n",
       "38    256\n",
       "39    215\n",
       "40    256\n",
       "41    250\n",
       "42    215\n",
       "43    253\n",
       "44    235\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "resp.age_r.value_counts().sort_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee9bc52d",
   "metadata": {},
   "source": [
    "We can use the `caseid` to match up rows from `resp` and `preg`.  For example, we can select the row from `resp` for `caseid` 2298 like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "f67bfd31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>caseid</th>\n",
       "      <th>rscrinf</th>\n",
       "      <th>rdormres</th>\n",
       "      <th>rostscrn</th>\n",
       "      <th>rscreenhisp</th>\n",
       "      <th>rscreenrace</th>\n",
       "      <th>age_a</th>\n",
       "      <th>age_r</th>\n",
       "      <th>cmbirth</th>\n",
       "      <th>agescrn</th>\n",
       "      <th>...</th>\n",
       "      <th>pubassis_i</th>\n",
       "      <th>basewgt</th>\n",
       "      <th>adj_mod_basewgt</th>\n",
       "      <th>finalwgt</th>\n",
       "      <th>secu_r</th>\n",
       "      <th>sest</th>\n",
       "      <th>cmintvw</th>\n",
       "      <th>cmlstyr</th>\n",
       "      <th>screentime</th>\n",
       "      <th>intvlngth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2298</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>902</td>\n",
       "      <td>27</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3247.916977</td>\n",
       "      <td>5123.759559</td>\n",
       "      <td>5556.717241</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>1234</td>\n",
       "      <td>1222</td>\n",
       "      <td>18:26:36</td>\n",
       "      <td>110.492667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 3087 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   caseid  rscrinf  rdormres  rostscrn  rscreenhisp  rscreenrace  age_a  \\\n",
       "0    2298        1         5         5            1          5.0     27   \n",
       "\n",
       "   age_r  cmbirth  agescrn  ...  pubassis_i      basewgt  adj_mod_basewgt  \\\n",
       "0     27      902       27  ...           0  3247.916977      5123.759559   \n",
       "\n",
       "      finalwgt  secu_r  sest  cmintvw  cmlstyr  screentime   intvlngth  \n",
       "0  5556.717241       2    18     1234     1222    18:26:36  110.492667  \n",
       "\n",
       "[1 rows x 3087 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp[resp.caseid==2298]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0164c24c",
   "metadata": {},
   "source": [
    "And we can get the corresponding rows from `preg` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "69de50aa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>caseid</th>\n",
       "      <th>pregordr</th>\n",
       "      <th>howpreg_n</th>\n",
       "      <th>howpreg_p</th>\n",
       "      <th>moscurrp</th>\n",
       "      <th>nowprgdk</th>\n",
       "      <th>pregend1</th>\n",
       "      <th>pregend2</th>\n",
       "      <th>nbrnaliv</th>\n",
       "      <th>multbrth</th>\n",
       "      <th>...</th>\n",
       "      <th>religion_i</th>\n",
       "      <th>metro_i</th>\n",
       "      <th>basewgt</th>\n",
       "      <th>adj_mod_basewgt</th>\n",
       "      <th>finalwgt</th>\n",
       "      <th>secu_p</th>\n",
       "      <th>sest</th>\n",
       "      <th>cmintvw</th>\n",
       "      <th>totalwgt_lb</th>\n",
       "      <th>totalwgt_kg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2610</th>\n",
       "      <td>2298</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3247.916977</td>\n",
       "      <td>5123.759559</td>\n",
       "      <td>5556.717241</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.8750</td>\n",
       "      <td>3.125000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2611</th>\n",
       "      <td>2298</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3247.916977</td>\n",
       "      <td>5123.759559</td>\n",
       "      <td>5556.717241</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.5000</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2612</th>\n",
       "      <td>2298</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3247.916977</td>\n",
       "      <td>5123.759559</td>\n",
       "      <td>5556.717241</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.1875</td>\n",
       "      <td>1.903409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2613</th>\n",
       "      <td>2298</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3247.916977</td>\n",
       "      <td>5123.759559</td>\n",
       "      <td>5556.717241</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.8750</td>\n",
       "      <td>3.125000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 245 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      caseid  pregordr  howpreg_n  howpreg_p  moscurrp  nowprgdk  pregend1  \\\n",
       "2610    2298         1        NaN        NaN       NaN       NaN       6.0   \n",
       "2611    2298         2        NaN        NaN       NaN       NaN       6.0   \n",
       "2612    2298         3        NaN        NaN       NaN       NaN       6.0   \n",
       "2613    2298         4        NaN        NaN       NaN       NaN       6.0   \n",
       "\n",
       "      pregend2  nbrnaliv  multbrth  ...  religion_i  metro_i      basewgt  \\\n",
       "2610       NaN       1.0       NaN  ...           0        0  3247.916977   \n",
       "2611       NaN       1.0       NaN  ...           0        0  3247.916977   \n",
       "2612       NaN       1.0       NaN  ...           0        0  3247.916977   \n",
       "2613       NaN       1.0       NaN  ...           0        0  3247.916977   \n",
       "\n",
       "      adj_mod_basewgt     finalwgt  secu_p  sest  cmintvw  totalwgt_lb  \\\n",
       "2610      5123.759559  5556.717241       2    18      NaN       6.8750   \n",
       "2611      5123.759559  5556.717241       2    18      NaN       5.5000   \n",
       "2612      5123.759559  5556.717241       2    18      NaN       4.1875   \n",
       "2613      5123.759559  5556.717241       2    18      NaN       6.8750   \n",
       "\n",
       "      totalwgt_kg  \n",
       "2610     3.125000  \n",
       "2611     2.500000  \n",
       "2612     1.903409  \n",
       "2613     3.125000  \n",
       "\n",
       "[4 rows x 245 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preg[preg.caseid==2298]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "315fbb82",
   "metadata": {},
   "source": [
    "How old is the respondent with `caseid` 1?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "d7dafb41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1069    44\n",
       "Name: age_r, dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "resp[resp.caseid==1].age_r"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d7c82ef",
   "metadata": {},
   "source": [
    "What are the pregnancy lengths for the respondent with `caseid` 2298?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "5c0a6ab2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2610    40\n",
       "2611    36\n",
       "2612    30\n",
       "2613    40\n",
       "Name: prglngth, dtype: int64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "preg[preg.caseid==2298].prglngth"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a205c407",
   "metadata": {},
   "source": [
    "What was the birthweight of the first baby born to the respondent with `caseid` 5013?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "5ae4cc07",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5516    7.0\n",
       "5517    NaN\n",
       "5518    8.0\n",
       "5519    8.0\n",
       "Name: birthwgt_lb, dtype: float64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "# One solution is the select all pregnancies with this `caseid` \n",
    "# and assume that the first valid value is the birthweight of\n",
    "# the first baby\n",
    "preg[preg.caseid==5013].birthwgt_lb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "976b0dbd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5516    7.0\n",
       "Name: birthwgt_lb, dtype: float64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Solution\n",
    "\n",
    "# A more complete solution uses `pregordr` to select the first baby\n",
    "\n",
    "preg[(preg.caseid==5013) & (preg.pregordr==1)].birthwgt_lb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dc5d50f",
   "metadata": {},
   "source": [
    "Examples and Exercises from Think Stats, 2nd Edition\n",
    "\n",
    "http://thinkstats2.com\n",
    "\n",
    "Copyright 2016 Allen B. Downey\n",
    "\n",
    "MIT License: https://opensource.org/licenses/MIT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2fdd906",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
